I recently needed to set up a VPN server that OS X would be able to connect to
without installing third-party software. The options looked to be IPSec/L2TP or
PPTP. With the scary warnings about
PPTP being
completely broken, I delved into the mildly terrifying world of IPSec/L2TP.
The web is full of pages telling you how to do everything IPSec related with
OpenSwan. However, OpenSwan needs out-of-tree kernel modules, which seem like
overkill to me (and a support headache), particularly when there's a working
implementation already in the Linux kernel. I decided to persevere with the
in-kernel NETKEY implementation of IPSec. I've successfully used this
implementation of IPSec numerous times before. Yet every single time I do, I
have ended up debugging through a process of downloading and reading source
code. I can see why many people choose to use OpenSwan! The debugging output
from racoon and the in-kernel implementation just isn't particularly
informative.
Below is a blow-by-blow account of how I got OS X talking to a IPSec/L2TP using
the in-kernel IPSec implementation. Hopefully this will help those who might
come here having Googled for the same error messages.
-
I used this
excellent guide on the Gentoo wiki, following the sections on
ipsec-tools and xl2tpd, and retrofitting for Ubuntu. To avoid repetition, I'm
not going to outline those steps again. In short, the Ubuntu-ified packages can
be installed using:
apt-get install ipsec-tools racoon xl2tpd
This got me most of the way, bar a few stumbling blocks described below. The
short summary is also now in the wiki page in the section on Mac OS X.
-
The first error I ran up against seems to be one of the most common from
racoon:
racoon: ERROR: no suitable proposal found.
racoon: [xx.xx.xx.xx] ERROR: failed to get valid proposal.
racoon: [xx.xx.xx.xx] ERROR: failed to pre-process ph1 packet (side: 1, status 1).
racoon: [xx.xx.xx.xx] ERROR: phase1 negotiation failed.
If you turn up the debugging sufficiently high on racoon (run it
from the command-line with racoon -vFdd), you get to see
what the desired proposals were. A number of different proposals were sent, but
this one was closest match to what I had specified in racoon.conf:
DEBUG: prop#=1, prot-id=ISAKMP, spi-size=0, #trns=6
DEBUG: trns#=6, trns-id=IKE
DEBUG: lifetime = 3600
DEBUG: lifebyte = 0
DEBUG: enctype = 3DES-CBC
DEBUG: encklen = 0
DEBUG: hashtype = SHA
DEBUG: authmethod = pre-shared key
DEBUG: dh_group = 1024-bit MODP group
The encryption type, hash type and auth method all match, but that
leaves the dh_group. It turns out that the setting of "14" in the
instructions refers to the 2048-bit MODP group, whereas here OS X is insisting
on using the 1024-bit MODP group.
Solution: Modify racoon.conf to use "dh_group modp1024".
-
This brings us to the next problem. Phase 1 negotiations begin,
but on the server, racoon was showing these errors:
racoon: NOTIFY: the packet is retransmitted by xx.xx.xx.xx[40276] (1).
And on the client side (in the Console application on OS X), these
messages appear:
racoon[17952]: IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
From the server's point of view, everything is completely normal, but the
client retries sending the same packet over and over. After some delving
through the source code of racoon on OS X, I tracked the cause to something
explained by this comment, near the code for checking the received identifier
(ipsec_doi.c:3689):
/*
* check the following:
* - In main mode with pre-shared key, only address type can be used.
...
Thus, the solution: The my_identifier setting in racoon.conf must be
an IP address, not an FQDN as described by the guide. It turns out that this is
actually racoon's default, so the simplest solution is just to delete the
my_identifier line completely from racoon.conf.
-
The final hurdle was figuring out how to allow clients to connect without
hardcoding the IP address of every client (i.e. allow "road-warrior" clients).
It turns out that there's an undocumented feature of racoon that allows you to
specify a hostname of "*" in psk.txt. For example:
* mysecretsharedpassword
Huzzah. Several hours of debugging later, a happy outcome. Here are the
relevant stanzas from racoon.conf, for reference.
remote anonymous {
exchange_mode main;
passive on;
generate_policy on;
nat_traversal on;
proposal_check obey;
proposal {
encryption_algorithm 3des;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group modp1024;
}
}
sainfo anonymous {
encryption_algorithm aes 256, aes, 3des;
authentication_algorithm hmac_sha1, hmac_md5;
compression_algorithm deflate;
}